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SYSTEM AND METHODFOR ASYNCHRONOUS STORAGE AND PLAYBACK OFASYSTEM STATE 

CROSS-REFERENCE TO RELATED APPLICATIONS 

This application claims the bcaiefit under 35 U.S.C. § 1 19(e) of U.S. Provisional 
5 AppHcation No. 60/395.236 filed July 1 1. 2002 and U.S. Provisional Application No. 

60/395,643 filed July 12, 2002. which appUcations are hereby incoiporated herein by reference 
in their entirety. 

STATEMENTS REGARDING FEDERALLY SPONSORED RESEARCH 
10 Not ^plicable. 

FIELD OF THE INVENTION 

This invention relates generally to computer systems, such as air 1ia£5c control systems, 
and, more particularly, to a system and method for staring and playing back a state of a computer 
15 syst^. 

BACKGROUND OF TEIE INVENTION 

As is Imown in the art, air trafBc control (ATQ is a service that promotes the safe, 
orderly, and eicpeditious flow of air trafSc. Safety is principally a matter of preventing 
colUsions with other aircraft, obstructions, and the ground, assisting aircraft in avoiding 
hazardous weather, assuring that aircraft do not operate in an airspace where operations are 
prohibited, and assisting aircraft in distress. Orderiy and expeditious flow assures the 
^ciency of aircraft operations along the routes selected by the operator. 



20 



25 



As is also known, air traffic control services are provided by air traffic control (ATC) 
systems. An air ttafSc control system is a type of computer and display system that processes 
data received fiom air surveillance radar systems for the detection and tracking of aircraft. Air 
traflSc control systems are nsed for bofli civilian and miUtary appUcations to determine the 
identity and locations of aircraft in a particular geographic area. 



30 
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It is desirable in aircraft control systems to store data recorded, obtained, or otherwise 
provided by. the air traffic control system. In particular, it is desirable to be able to store and 
play back display data, including images shown on the display system of the air traffic control 
system. It is desirable to be able to record and play back air traffic control data in case of 
5 acddait, for teaching puiposes, fbr simulation purposes and the like. 

Conventionally, there are several ways to record ATC display data. One technique is to 
use a video recorder to store ATC display data at each ATC display station. While tWs 
approach does not interrupt system operation (i.e., it has no impact on computer code or system 
10 performance and thus is a transparent recording technique), the Video recorder approach has 
several drawbacks. For example, there is a relatively large amount of display data to store. If 
videotapes are used for storage, a relatively large number of videotapes are required to store all 
of the necessaiy display data associated with a period of time (e.g., 24 hours). Also, there are 
typically multiple ATC stations^tenninals at which storage is desired. Thus, multiple video 
• 15 recorders or other recording devices are required. Furthertaore, it is relatively time consuming 
to locate a specific location on a videotape during playback of the display data. 

Another technique for tranqjarendy storing data is to provide computer code used in 
ATC ^sterns having instructions for sending messages to a storage device, wbich identify to 
20 the storage device the operations being performed by the ATC system. The messages can be 
sent to the storage device either before or after a conesponding operation is performed. To 
playback what has oocinred in the ATC system, the storage device sends the stored messages to 
the ATC ^stran and the ATC display system carries out the messages. 

2 5 However, if the developer of the computer code feils to include code to store a certain 

step or operation, or if the developer fails to inchide code to playback a certain message, then 
the record or playback wiU not be accurate. Another problem with this approach is that it is 
necessary to process a relatively large amount of data. Also, it is very expensive in terms of 
code development because it is very time consuming to include all of the additional computer 

3 0 code to store and play back ihe operations being displayed on the display system. 

2 
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It would, therefore, be desirable to overcome the aforesaid and oHier disadvantages. 

SUMMARY OF THE INVENTION 
5 A system for asynchronous storage and playback of a system state provides 

"instantaneous images'" of a system without interfering with the system behavior and response 
time. The images are associated with an ordered set of commands referred herein as a 
"dynamic snapshot," When stored and later executed in the proper order, the dynamic snapshot 
commands set the system to a snapshot state, which is representative of a time at which the 
1 0 dynamic snapshot was recorded. 

In one particular embodiment, the system &r asynchronous storage and playback of a 
system state includes a storage device for storing at certain times an image or ^'snapshot'* of an 
air traffic control (ATC) display. The system can also store changes that occm* on tfie display 

1 5 between the snapshots. With this paiticolar airangement, a system for transparently storing the 
ATC display is provided that does not interfere with otiier processing performed by fte ATC 
system. By storing the display at certain times (i-e. storing the display snapshot), and then 
recording the changes on the display which occur between Ae certain times, the system records 
a reduced amount of data than is recorded in prior art approaches, without a decrease in the 

20 accuracy of the recording. While the systan is described ia association with an ATC display, it 
should be recognized that tibie system is applicable to any system having software commands or 
instructions. 

In accordance with the present invention, a method of storing commands includes 
2 5 recording a first set of commands to a command quwe as a first dynamic snapshot and storing 
the &st dynamic snapshot. The method also includes recording one or more additional sets of 
commands to the command queue and eliminating overridden commands from the command 
queue to provide a second dynamic snapshot The second dynamic snapshot is stored. In one 
embodiment, the commands ate display commands associated with an ATC display. 

30 
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In accordance with another aspect of the present invention, stored commands associated 
with a time of interest are played back by receiving the time of interest and retrieving a stored 
dj^amic snapshot corresponding to a time at, or prior to, the time of interest Additional sets of 
stored commands are also retrieved and appended to the dynamic snapshot to provide an 
intermediate dynamic snapshot. The iirtennediate dynamic snapshot is intopreted. In one 
particular embodiment, the stored dynamic snapshot includes display commands associated 
with an air traffic control (ATC) display and the intexpreting results in a view of die ATC 
display corresponding to the time of interest In one particular embodiment ovaridden 
commands within the intemiediate dynamic snapshot are eliminated, ' 

In accordance with another aspect of the present invention, a computer program 
medium for storing commands includes instructions for recording a first set of commands to a 
command queue as a first dynamic snapshot and instructions for storing the first dynamic 
snapshot. The computer program medium also includes instructions for reconJing one or more 
additional s^s of commands to the command queue and instructions for eliminating overridden 
commands fix)m the command queue to provide a second dynamic snapshot. The computer 
program medium also includes instructions for storing the second dynamic snapshot. In one 
embodiment, the commands are display commands assodated with an ATC dis^^ 

In accordance with another aspect of the present invention, a computer program 
medium for storing commands includes instructions for receiving a time of interest and 
instmctions for retrieving a stor&d dynamic snapshot corresponding to a time at, or prior to, the 
time of interest The computer program mediiun also includes instructions for retrieving 
additional sets of stored commands and instructions for appending the additional sets of stored 
25 commands to the dynamic snapshot to pre vide an intermediate dynamic snapshot The 

computer program medium also includes instructions for interpreting the intexmediate dynamic 
snapshot In one particular embodiment, the stored dynamic snapshot includes display 
commands associated with an air traffic control (ATC) display and the commands for 
interpreting result in a view of the ATC display corresponding to Ae time of interest. In one 
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particular embodiment, the computer program medium includes instructions for eliminating 
overridden commands from within the intennediate dynamic snapshot 

In accordance with another aspect of the present invention, a system includes a 
5 recording proxy, a dynamic snapshot generator, a command interface, and a storage device, all 
coupled so as to store dynamic snapshots and additional display commands in the storage 
device. 

With this invention, the dynamic snapshot can be recorded and stored without 
10 ■ intenrupting real-time system operation. In one embodiment, this approach allows a display 
image to be stored without an impact on the user, i.e., without teezmg a real-time display and 
also without burdening a computer code developer by requiring the developer to write 
additional computer code to record and/or store system oommaads. The present-invention 
allows the asynchronous storage and playback of a stored system state (snapshot) without 
15 impacting System real-time operation. 

The present invention finds application in display recorders (e.g., of the type used in air 
traffic control systems) as well as in systans (e.g., database systems) in which it is desirable to 
be able to rapidly recreate a data set without interfering with real-time system behavior (e.g-, 
20 response time). 

BRIEF DESCRIPTION OF THE DRAWINGS 

The foregoing features of this invention, as well as the invention itself, may be more 
fially understood from the following description of the drawings in which: 

25 FIG. 1 is ablock diagram of an airtrafSc control (ATC) recording system, which utilizes a 

dynamic snapshot and has store and playback capability; 

FIG. 2 is a diagram illustrating formation of a dynamic snapshot; 

FIG. 3 is a block diagram of a dynamic snapshot command queue; 

FIG. 4 is a diagram illustrating storage of dynamic snapshots and storage of 
3 0 intennediate sets of commands; 



5 



PA6E69»2'RCVDAT7/7/20063:55:28PM[EasternD *DURAnON(ninKS):3(l-20 



87/07/2006 15:04 7814019966 DCM, LLP PAGE 



FIG. 5 is a flow diagram illustrating a set of processing steps to generate and to store a 
dynamic snapshot; 

FIG. 6 is a flow diagram illustrating a set of processing steps to eliminate ovenidden 
commands from a dynamic snapshot; 
5 FIG. 7 is a flow diagram illustrating a set of processing steps used to provide a playback of 

stored dynamic snapshots and stored intermediate sets of commands; and 

FIG. 8 is a flow diagram illustrating an alternate set of processing steps used to provide a 
playback of stored dynamic snapshots and stored intenaiediate sets of commands. 

10 DETAILED DESCRIPTION OF THE INVENTION 

Before describing a system and method for asynchronous storage of a system snapshot, 
some introductory terms are described- As used herein, the terms **data storage" and 
"recording" are -used synonymously to refer to retention of data. However, as used herein, the 
term "storage" can, in some embodiments, refer to retration of data fbr a particularly long 

15 period of time, for sample, hours or days. 

A scene graph will be understood to be a particular representation containing 
information about the geometry and appearance of objects appearing on a graphical display. 
The scene graph is a dynaioDdc data structure within a computer program that can also be saved 
20 as a file. The scene graph includes data that describes shape objects (geometry and 

appearance), geometric structure relationships (geometric transfortnations, ordering, and 
grouping), global objects Oiow all shape objects are viewed, e.g., viewpoints, lights, 
backgrounds), behaviors (procedures for modifying infomation stored in a scene graph), and 
the like. 

25 

Tlxe scene graph is object oriented, having software objects that describe the shape 
objects and software commands that perform the behaviors upon the shape objects. For 
example, a scene graph can include a software object associated with an aircraft image, and a 
scene graph display command can operate on the aircraft object to render the aircraft image on 
30 a graphical display. 

6 
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The objects of a scene graph are created using software commands, for example a 
"create" conraiand. The objects of a sc^e graph are operated upon using othar commands, for 
example a "render" command, which causes an object to appear as an image on a Addeo screen. 
5 Therefore, the scene graph, inchiding the objects, is associated with a set of scene gr^h 
display commands. 

A scene graph can be represented diagrammatically as a tree structure having "nodes" 
and interconnecting lines or '"arcs," The scene graph data structure described above underlies 
10 the tree structure representation. 

As used herein, the term "scene graph" is used to describe the underlying data structure 
associated with a scene graph, as opposed to the set of scene graph display commands or the 
• scene graph tree structure. 
15 ' 

It should be understood that a scene grsph can be associated with more scene graph 
display commands than actually are used to generate images on a graphical display. For 
example, a scene .graph can be associated with a set of ''create" commands that represent scene 
graph objects, and not every object necessarily has a corresponding "render" command that 
2 0 generates an image on the graphical display. 

Various hi^-level software application programmer interfeces (APIs) have been 
established to create a scene graph when presented with the scene graph display commands. 
For example Java3D and VRML provide high-level software to generate a scene graph. Lower 
2 5 level APIs have also been provided, including Open GL, and Direct 3D. 

Scene graph tedmiques are conventionally used to provide a scene graph associated 
with three-dimensional images on a graphical display, for example in computer games. To this 
end, hardware manufacturers have provided three dimensional (3D) graphics dicuit boards, 
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having local processmg capability on the graphical circuit board, and having the ability to 
interpret scene graph data and providing a conpesponding graphical display on a monitor. 

Scene graph progranuning techniques, in conjunction with the 3D graphic circuit board. 
5 provide the ability to rapidly render a 3D image on a graphical display with minimal impact on 
a central computer processor. Images on the graphical display can also be rapidly updated with 
one or more display commands, interpreted by an API, and sent to the 3D graphics drcuit 
board. 

While existing scene graph APIs provide three-dimensional (3D) graphical objects and 
corresponding 3D images on a graphical display, a conventional air traffic control (ATQ 
display provides two-dimensional (2D) images* Two-dimensional images are conventionally 
provided without use of scene graphs and without taking advantage of the local processing 
capability of the 3D graphics circuit board. Instead, 2D images are conventionally generated 
with APIs that can interpret a very low-level "paint" command. A single paint command is 
able to render a simple shape, for example a line. Therefore, in order to render more conqslex 
shapes on an ATC di^lay, such as aircraft and geographic features, numerous paint commands 
are conventionally generated, which are then interpreted by a low level API in conjunction with 
an ATC central processor. Therefore, the convaitional ATC display requires substantial 
processing time provided by the ATC central processor, in order to process the large number of 
'"paint" commands. 

However, a 2D scene graph technique suitable for use in an ATC display system is 
described in U*S. Patrat Application entitled "Scene Grsqph Based Display for Desktop 
25 Applications," filed on July 11, 2003, and assigned Application No. 10/617,599,[ which is 
assigned to the assignee of the present invention and incorporated herein by reference. With 
the 2D scene graph technique, the ATC display images can be associated with two-dimensional 
(2D) scene graph display commands, which are interpreted by a 2D scene graph API to provide 
a 2D scene graph. The 2D ATC display can be updated with 2D scene graph display 
3 o commands. 

8 
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As used herem, the term "snapshot" or "dynamic snapshot*' should be broadly constraed 
to refer to a set of system parameters describing a system "state" at a particular time. In one 
particular embodiment, the snapshot corresponds to a complete set of all system parameters 
describing the system state at the particular time. For example, in one particular embodiment, 
the dynamic snapshot is a snapshot of display commands representative of all images on a 
grgphical display. It should be recognized that, whereas the exemplaiy dynamic snapshot 
includes the display commands themselves, tiie scene graph instead includes scene graph data 
generated by an API in response to scene graph display commands. 

While the present invention is described in terms of scene graphs and scene graph 
commands below, and in particular in relation to an ATC display using 2D scene graphs, it 
should be appreciated that the present invention is also applicable to graphical display images 
using conventional "paint" commands and to systems other than ATC display systems. 



Referring now to FIG, 1 , a system 1 0 for asynchronous recording of a systan snapshot 
includes a radar system 12. The radar system 12 provides radar information to a software agent 
14, The radar information can include, for exan^le, range, elevation, and azimuth information 
associated with one or more airc^fL The radar information can also include, for esmnple, 
2 0 geographic information such as land topology^ including mountains and hills. 

The software agent 14 interprets the radar information and provides display commands 
16. In one particular embodiment, the display commands are scene graph display commands 
including commands representative of objects and of renderings. A recoding proxy 18 

2 5 forwards the display commands 1 6 to a real-time ATC display system 20 wifli no noticeable 

delay. 

A user interface 15 allows a user to provide inputs to the software agent 14, for example 
as mouse scrolls, mouse clicks, or the like, for example, to scroll, to zoom, and to otherwise 

3 0 interact with images presaited on a monitor 26. The user actions are interpreted by the 
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software agent 14 to generate additional display commands 16, 

A real-time ATC display system 20 includes a central processing unit (CPU) 23 
operating in conjunction with an API 2 1 . The display commands 1 6 are received by the API 
5 21 y which operates upon the conamands to generate a scene graph 24 stored within a graphics 
module 22. In one particular anbodiment, the graphics module 22 is a three-dimensional (3D) 
graphics circuit board having a local processor (not shown) and capable of storing the scene 
graph 24. The graphics module 22 is coupled to a monitor 26, which provides an ATC display 
assodated with the scene graph 22 and widi the display commands 16- 

xo 

It should be recognized that the system 1 0 has relatively few display commands mjing 
the scene graph techniques. In another «nbodiment using the "paint'* display commands, the 
system 10 has more display commands. Therefore, the central processing unit (CPU) 23 of the 
real-time display syst^ 20, spends less time on processing the scene graph display commands 
15 16 than it would in processing *'paint " display commands. 

The recording proxy 1 8 also provides the display commands 16 to a dynamic snapshot 
generator 30 and to a display command interfuse 28. The dynamic snap shot generator 30 
captures a system snapshot^ Le., a dynamic snapshot, associated with the real-time display 

2 0 system 20. In so doing, the dynamic snapshot generator 30 captures display commands 

associated with images presented on the ATC monitor 26 at a particular time. In one particular 
embodiment, the dynamic snapshot generator 30 captures display conamands associated widi all 
such images* In capturing the dynamic snapshot, the recording proxy 1 8 may oppend a time 
stamp to one or more particular display comtnands. Use of the time stamps will become 
25 apparmt in association with FIGS. 7 and 8 below. In operation, the dynamic snaqpshot can be 
minimized in size using techniques described in conjunction with FIG. 3 bdow. 

For reasons described above, the dynamic snapshot generated by flie dynamic snapshot 
generator 30 is not the same as the seme graph 24. The scene graph 24 contains processed 

3 0 display commands (processed by API 21) and the dynionic snsi^hot contains unprocessed 

10 
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display commands 16, The display commands included in a particular dynamic snapshot are 
representative of those objects and roaderings appearing as images on the monitor 26 at a 
particular time. 

5 The recording proxy 1 8 also provides the display commands 1 6 to the display command 

interfece 28. The display command intaface 28 captures one or more display commands 16 as 
tfaq^ are provided by the software agent 14. Generally, the set of commands captured by the 
display command interfecc 28 is a relatively small set of display commands, which does not 
represent an entire dynamic snapshot Instead, the set of commands captured by the display 
1 0 command interface 28, if presented to the real-time display system 20, would provide an update 
of one or more existing ATC display images. 

A dynamic snapshot 32 provided by the dynamic snapshot generator 30 is stored by a 
storage device 36. Subsequent dynamic snapshots 32 are stored to the storage device 36 at first 
15 predetermined intervals th^eafter. A set of display commands 34 provided by the display 
command interface 28 is also stored to the storage device 36. Subsequent sets of display 
commands 34 are stored to the storage device 36 at second predetOTnined intervals thereafter. 
The stored dynamic snapshots 32 and the stored sets of display commands 34 are herein 
referred to as "stored data." The sequence of storage to the storage device 36 is described in 

2 0 greater detail in conjunction with FIG. 4 below. 

In one particular embodiment, the storage device 36 indudes a tape media capable of 
storing digital data. In another embodiment, the storage device 36 is a solid-state storage 
device, for example a non-volatile solid-state storage device. 

25 

It should be appreciated that the dynamic snapshot generator 30 can provide the 
dynamic snapshot 32 to the storage device asynduronously from display activities of the real- 
time display system 20. It should also be appreciated that the display command intcrfece 28 
can provide the sets of di^Iay commands 34 to the storage device 36 asynchronously fiom the 

3 o display activities of the real-time display systOTi 20. Once recorded, the dynamic snapshots 32 

11 
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and the sets of display commands 34 contain infonnation necessaiy to later reconstruct images 
earlier seen on the monitor 26. 

Upon playback, the recorded sets of display commands 38 are provided to a display 
5 command interface 42. Also, the recorded dynamic snapshots 40 ai^ provided to a dynamic 
snapshot generator 44. During playback, in one embodiment, the dynamic snapshot generator 
44 can provide a dynamic snapshot that is minimized in size as described below in conjunction 
with FIG. 3. 

10 Two types of playback of the recorded data are described in conjunction with FIGS, 7 

and 8 respectively. Let it suffice here to say that a recorded dynamic snapshot 48 and one or 
more recorded sets of display commands 46 are provided to a playback ATC di^lay system 50 
having an API 51 associated with a CPU 53, a graphics module 54, and a monitor 56, Upon 
playback, the dynamic snapshot provided by the dynamic snapshot generator 48, when 

15 interpreted by the API. 5 1 , provides a scene graph 54. The scene graph 54 is interpreted by the 
graphics module 52 to provide corresponding images on the monitor 56 , wfaidi, in one 
embodiment (e.g., FIG. 7)^ can correspond to a time of interest. However, in another 
embodiment (e.g., FIG. 8> , the scene graph 54 can be associated with a time before the time of 
interest and the sets of display commands 46 can bring the scene graph 54 forward in Hme to 

20 the time of interest. 

It should be imderstood that Ihe images fbvs presented on the monitor 56 correspond to 
those images seen at the earlier time of interest on the monitor 26, part of the real-time display 
system 26. Therefore, a user can view what was displayed earlier. 

25 

In operation, the display commands 1 6 received by the graphics module 22, provide the 
scene graph 24 as weU as updates to the scene graph 24. As desoibed above, a scene graph can 
be represented by a processed group of display commands and an update to the scene graph can 
be represented by oih&c processed display commands. For example, an image of a particular 
3 0 aircraft presented on the monitor 26 can cozrespond to one or more processed display 

12 
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commands that invoke an aircraft geometric object from the scene graph 22. Also, a change of 
characteristics of the particular aircraft, for example position, can correspond to one or more 
other processed display commands that change the invocation of the aircraft geometric object 
from the scene graph 22, while other geometric objects presented as images on the monitor 26 
5 rraiainunchanged* 

Referring now to FIG. 2, a first dynamic snapshot 1 00 is obtained at time Iq. The first 
dynamic snapshot 100 is combined with commands 102 recorded between time to and some 
later time designated ti to provide a second dynamic snapshot 104 associated with the time t(. 
10 The second dynamic snapshot 1 04 includes display commands associated with the time ti, and 
tiherefore contains information about the state of images on the monitor 26 at the time ti. 

• Using the inventive scene graph technique, a snapshot of the ATC display system can 
' be represented using relatively few commands. The relatively few commands, however, 
15 conrespond to images on the ATC display at a particular time. It is possible to "move" the 
dynamic snapshot forward through time by adding display commands, without affecting the 
operation of the system 1 0 (FIG. 1) in a substantive way. 

It should be appreciated that retrieving a dynamic snapshot from the storage device 36 

2 0 (FIG. 1 ) is an iterative process. That is, the process of retrieving the dynamic snapshot begins 

from a "known" system state, which is associated with a particular dynamic snapshot. Hie 
dynamic snapshot is then moved forward in time to a time of interest by adding subsequent 
recorded display commands to the dynamic snapshot. 

25 Referring now to FIG, 3, an exemplary command queue 120 is contained in the 

dynamic snapshot generators 30, 44 (FIG, 1). The command queue 120 includes a dynaimc 
snapshot portion 122 and a command accumulation portion, also refeired to as a command 
stadc 124. Once a recording of a particular dynamic snapshot 122 begins, commands issued 
after that time are accumulated in the command stack 124 until the recording is completed, and 

3 0 until the time that a next dynamic snapshot is recorded. 

13 
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Shortly before tiie time that the next dynamic snapshot is stored in the stomge device 36 
(FIG- 1) the dynamic snapshot 122 is updated to a state then coiresponding to the ATC display 
system 20 (FIG. 1). The dynamic snapshot is updated by appending the conraiand stack 124 to 
5 the dynamic snapshot 122, to become the next dynamic snapshot It should be understood that, 
without further processing, the dynamic snapshot 122 would progressively grow in size. 
Therefore, overridden, redundant, and/or superfluous commands can be removed fix>m the 
command queue 120 to provide a dynamic snapshot 122 that is reduced in size. 

10 It is understood that in general, an ovendding display command is a display command 

that reverses an action of an earlier issued oveixidden display command A redundant display / 
command is a display command that provides the same function as earlier issued display 
command. A superfluous display command is a display command that has no function in a 
given context. The removal of the overridden, redundant, and/or superfluous commands is also 

1 5 described in conjunction with FIG. 6, 

In one embodiment, the syst^ determines if any (and which) di^lay commands within 
the cotmuand stack 1 24 override display commands within the dynamic snapshot 122. The 
overridden commands are removed &om the dynamic snapshot 122 and from the command 
20 stack 124. 

In order to remove overridden display commands, in one embodiment, display 
commands are translated or brokra down into new sequraces of display commands. The 
following show various examples of commands, which are broken down into new sequences. 

25 

The following are examples of two display commands: 

group.add(index, element) // Inserts the specified element at the 

// specified position in the group, 
30 // Shifts the elements cunenfly at that 

// position (if any) and any subsequent 
// dement to theright (adds one to 

14 
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//their indices). 



10 



group.remove(index) 



// Removes the element at the specified 
// position in this group. Shifts any 
// subsequent element to the left 
// (subtracts one firom their indices). 



It is not easy to detennine if a remove display command overrides an add display 
command. Indeed, betwe^ add and remove display commands, other 'insertions" and/or 
'"temovals*^ may have been perfonned and the elements could have shifted. Several options can 
resolve this uncertainty* 



15 



A first option converts "add" and "remove" display commands to sequences of "set" 
commands, e-g.^ group,5et(ekment, index) display coromands. For example: 



groiq>: 


A 


. B 


C 


D 


null 


null 




0 


1 


2 


3 


A 


5 



group.add(1, E): 



group.set(4, D) 
group.s^(3, C) 
group.set(2, B) 
group.set(1, E) 



group; 



group: 



A 


B 


C 


b 


null 


null 



group.remove(l): 



group. set(1, C) 
group.set(2» D) 
group.set(3, null) 



A 


E 


B 


C 


D 


null 


group: 


A 


C 


D 


null 


null 


null 


0 


1 


2 


3 


4 


5 




0 


1 


2 


a 


4 


5 



With this particular option, two "set" conmiands override each other if and only if they are 
performed on the same group having the same index. 



20 



A second option breaks down the group into group nodes (linked list) and replaces add and 
remove commands with sequences of createGroupNodeQ^ groupNode.setNext(groupNode), 
groupNode.set£lement(elemait) and groupNode.disposeO commands^ For example: 



15 
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group: 



A 


B 


C 


D 


group: 


A 


B 


C 


D 


0 


1 


2 


3 




0 


1 


2 


3 



group.add(1, E): 



n :b createGroupNodeO 
n.setNext(1) 
n.setElement(E) 
O.setNext(n) 



group.remove(l): 



0. 5etNext(2) 

1. dispose() 



group: 


A 


E 


B 


C 


D 


group: 


A 


C 


D 




0 


' n 


1 


2 


3 




0 


• 2 


3 



The second option is usually fester than the first option described above during 
recording but requires morp processing than the first option during playback (to recreate the 
group from the group nodes). 



10 



In one pardcular embodiment, for each new command applied to the command stack 
124, the command queue 120 is examined in order to remove not only the overridden 
commands, but also redundant and superfluous display commands. The steps performed in 
order to remove the overridden, redundant, and superfluous display commands arc described in 
conjunction with FIG. 6. 



In one embodiment, by removing overridden, redxmdant, and superfluous commands 
fonn the command queue 120, the size of the corrmiand queue 120 can be maintained at or near 
15 about one hundred kilobytes. 



Referring now to FIG. 4, a chart 150 showing storage of display commands includes a 
time scale 164 having times and time intervals. A first set of display commands 152. 
conesponding to a dynamic snapshot, is stored at time tO. The dynamic snapshot 152 can 
20 correspond, for example, to the dynamic sn^ahot 122 of FIG. 3 (also 32, Fia 1) and the 

storing can be provided, for example, by the storage device 36 of FIG. 1 . As described above, 
the dynamic snapshot 152 includes display commands associated with images on an ATC 

16 
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10 



display at the time tO. There can be any number of display commands in the dynamic snapshot 
152. 



After a time interval Tl, a set of display cottimands 154 is stored. The set of display 
commands 154 can correspond, for example, to the set of display commands in the display 
command interface 28 of FIG. 1, and the storing can be again provided^ for example, by the 
storage device 36 of FIG. 1 . The set of display commands 1 54 can have any number of display 
commands, however, the set of display commands 154 generally has fewer display commands 
than the dynamic snapshot 152. 



After a further time interval T2„ a set of display commands 156 is stored. The set of 
display commands 156 can correspond, for example, to the another set of display commands in 
the display conmiand interface 28 of FIG, 1 , and the storing can be again provided; for 
example, hy the storage device 36 of FIG. 1. The s^ of display commands 156 can have any 
1 5 number of di^Iay commands, including a different number of display commands than the set 
of display commands 154. However, the set of display commands 156 generally has fewer 
display commands than the dynamic snapshot 152. 

Odier sets of display cotmnands, for example a set of display commands 160^ are 
2 0 similarly stored. The time intervals Tl, T2 and other time intervals associated witfi other ones 
of the sets of display commands, e.g., 1 54, 156, 158 can be liie same time intervals or, in oAer 
embodiments, can be different time intervals. 

At a time tl , anottier dynamic snapshot 1 60 is stored. The dynamic snapshot 1 60 can 
25 conespond, for example, to another of the .dynamic snapshots 122 of FIG 3 (32, FIG. 1) and tiie 
storing can be provided, for example, by the storage device 36 of FIG. 1. As described above, 
liie dynamic snapshot 160 includes commands associated with images on an ATC di splay at the 
time tl. There can be any number of display cormnands in the dynamic snapshot 1 60, and the 
dynamic snapshot 160 need not have the same number of display commands as the dynamic 
30 snapshot 152. 



17 



PA6E81/92'RCVDAT7m 3:55:28m[East^ 'DUMTION(innKS):30-20 



07/07/2006 15:04 7814019966 



DCM, LLP 



PAGE 



The time interval between the time tl and the time tO is a time interval TM, which is 
larger than any one of die tiiue intervals Tl-TN. Therefore, the dynamic snapshots, for 
example, the dynamic snapshots 152, 160, are stored ftom time to time and the sets of display 
5 commands 156^ 158, 160 are stored more often. 

It should be appreciated that given the stored dynamic snapshot 152, and given the 
stored sets of display commands 156, 158, 160, which together form the stored data within the 
storage device 36 of FIG, 1 , during a playback of the stored data, the stored dynamic snapshot 

10 152 can essmtially be moved forward in time by appending one of more of the stored sets of 
display commands 154, 156, 158 to the dynamic snapshot 152 to provide a more recent 
dynamic snapshot Also, in one particular embodiment, upon appending the one or more sets 
of display commands 154, 156, 158 to the dynamic snapshot 152, ovmidden, redundant^ and/or 
superfluous display commands can be removed &om the resulting dynamic snapshot in much 

15 the same way as described above for recording in conjunction with* FIG. 3. 

In one particular embodiment, the time intervals Tl, T2, TN are equal in duration and 
less than. one second- In another embodiment, the time intervals Tl, T2, TN are equal and less 
than five seconds. In another embodiznent, the time intervals Tl , T2, TN are equal and less 
2 0 than sixty seconds. However, in other embodiments, time intervals Tl , T2, TN correspond to 
other times and may or may not be equal. 

In one particular embodiment, the time interval TM is equal to time intervals between 
storage of others of the dynamic snapshots (not shown) and is greater than sixty seconds. In 
2 5 another embodiment, tiie time interval TM is greater than five minutes. In another 
embodiment, the time interval TM is greater than ten minutes. However, in other 
embodiments, time intervals TM corresponds to other times and may or may not be equal to 
time intervals between recording of other dynamic snapshots. 

30 It should be appreciated that longer intervals between recoiding of dynamic snapshots 

18 
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results in less recording bandwidth, but a longer seeking time upon playback to find a time of 
interest- Shorter time intervals, tha-efore, result in greater recording bandwidth and shorter 
seeking times. 

5 In one embodiment the techniques of the present invention have been used to 

demonstrate that (I) recording does not significantly impact rendering performance (less than 
2% of ATC central processor usage) and can be performed by background threads; (2) 
record/playback bandwidth is relatively low (the sets of display commands 154, 156, 158 
represent infrequent changes to the dynamic snapshots, e.g. 152, 160, which are taken, for 
10 example, every 5 minutes); and (3) seeking a time of interest during a playback, i.e., to move a 
dynamic snapshot forward in time, can be relatively fast Oess than 1 second for up to 5 minutes 
of time movement)^ 

• FIGS. 5-8 show process steps used to record and playback dynamic snapshots and sets 
15 of display commands, for example the dynamic snapshots 1 52, 1 60 and the sets of display 
commands 154, 156, 160 of FIG. 4, 

Rectangular blocks of FIGS. 5-8 correspond to software processing steps associated 
with a software command or a set of software commands^ and diamond blocks represent 

2 0 software decisions. Alternatively, the processing and decision blocks represent steps 

performed by ftmctionally equivalent circuits such as a digital signal processor circuit or an 
application specific integrated circuit (ASIC). The flow diagrams do not depict the syntax of 
any particular programming language. Rather, the flow diagrams illustrate the fimctional 
information one of ordinary skill in the art requires to &bricate circuits or to generate computer 

2 5 software to perform the processing reqxiired of the particular apparatus. It should be noted that 
many roxitine program elements, such as initialization of loops and variables and the use of 
temporary variables are not shown. It will be appreciated by those of ordinary skill in the art 
that unless otherwise indicated herein, the particular sequence of steps described is illustrative 
only and can be varied without departing fi^m the spirit of the invention. 

30 
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Referring now to HG. 5, a flow chart 200 represents a series of steps used to iword 
the dynamic snapshots 152, 160 and the sets of display commands 154, 156, 160 of FIG. 4. 
Processing begins at step 202, at which a point in time is selected at which recording will 
begin (referred to as a record point). The record point is typically associated with a recorded 
5 time stamp. The record point is a point at wJiich at least a set of display commands 

corresponding to a dynamic snapshot has been acquired in the command queue, for exanq>le, 
the command queue 120 of FIG. 3. 

At step 204, a jSrst set of display commands is acquired and temporarily recorded in a 
1 0 solid-state memory or the like. The first set of display commands, as described above, 
corresponds to at least a set of display conunaods corresponding to adynamic sn^shot 

Overriddea commands are eliminated at step 206 fit>m tibe command queue, for 
example the conunand queue 120 of FIG. 3, after which the process continues to step 208, 
15 where tiie first dynamic snapshot is stored, for exaiv^le to the storage device 36 of FIG. 1 . an 
alternate embodiment, redundant and/or superfluous commands are also removed at step 202 
(see FIG. 6)- As this is the first dynamic snapshot, there may be no overridden, redundant, or 
superfluous commands in the command queue. At step 208, the dynamic $n^$hot then stored 
is not deleted firom the command queue 120. 

20 

Additional display commands are acquired and recorded to the command stack at step 
210, for example, to the command stack 124 of FIG. 3. It will be appreciated that some of the 
additional display commands can be recorded in the command stack during the time that the 
dynamic snapshot is being stored to the storage device at step 208. Therefore, the storage of 
25 the dynamic snapshot can be perfomied asynchronously fixmi the acquisition of additional 
display commands at step 210. Furthermore, the storage of the dynamic snapshot at step 208 
can occur asynchronously firom other aspects of the display processing. 

At step 212, a decision is made as to whether it is time to store another dynamic 
30 snapshot, i.e., whether a time interval TM has elapsed since storage of the last dynamic 

20 



PAGEH«2'RCVDAT7m63:55:28PM[EasternD^^^ * DURATION (innKS):3l)-20 



07/87/2086 15:04 781401996B 



DCM, LLP 



PAGE 85/92 



snapshot Time intervals associated with storage of the dynamic snapshot are discussed above 
in conjunction with FIG. 4. When enough time has elapsed, the process proceeds to step 214. 

Overridden commands are eliminated from the command queue at step 214. As 
5 described in conjunction with FIG, 3, the oveiridden conmiands can be eliminated, for 
example, from among the command stack 124 and the dynamic snapshot 122, to generate a 
new dynamic snapshot. In an alternate onbodiment, redundant and/or superfluous commands 
are also removed at step 214 (see FIG. 6). 

10 At step 216, the new dynamic snapshot is stored to the storage device 36 (FIG. 1), after 

which, at step 218, a decision is made as to whether a total storage time has elapsed, A total 
storage time can be associated with, for example, a fiiU storage device 36. If the total storage 
time has not elapsed, then tfie process returns to step 210, where additional display commands 
are accumulated to provide and store yet another dynamic snapshot at step 216. 

15 ' 

At step 220, the additional display commands are also acquired, i.e., recorded in a 
display command interface, for example, the display command interface 28 of FIG. 1 . 

A decision is made at step 222 as to whether it is time to store the additional disfplay 
2 0 conmiands, i,e., whether a time interval Tl has elapsed- Time intervals associated with storage 
of the additional display commands are discussed above in conjunction with FIG. 4. When 
enough time has elapsed, the process proceeds to step 224, at which time the additional display 
commands, for example, those additional display commands accumulated in the display 
command interface 28 (FIG. 1), are stored in the storage device 36. Upon storage of the 
2 5 additional display commands at step 224, the additional display commands can be deleted from 
the display command interface 28. 

Like the dynamic snapshots stored at steps 208, 2 16, it will be appreciated that some of 
Ae additional display commands can be recorded at step 220 in the display command interfece 
30 28 during the time that the additional display commands are being stored to the storage device 

21 
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al step 224. Therefore, the storage of the additional display commands can be perfoimed 
asynchronously from the acquisition of additional display commands at step 220. Furthermore, 
the storage of the additional display commands at step 224 can occur asynchronously from 
other aspects of the display processing. 

5 

At step 226, a decision is made comparable to the decision of step 218 described above, 
where it is determined whether the recording should be tenninated. If the recording is not 
taminated, the process retums to st^ 220, where the earlier additional display commands are 
erased and new additional display commands are recorded. 

10 

With the process 200, the dynamic snapshots are stored at steps 208 and 216 with a time 
interval TM, while the additional display commands are stox«d at step 224 with a time interval 
TL G^CTally the time interval Tl is less than the time int^ral TM. In other words, the 
dynamic snapshots are stored with longer time intervals between dynamic snapshots and the 
15 additional display commands are stored with shorter time intervals between sets of the 

additional display commands. In an alternate embodiment, the time interval Tl and/or TM can 
be varied at each cycle beginning at the decisions 226, 228 respectively. 

Refening now to FIG. 6, a iM:ocess 250 for removing ovenidden, redundant, and/or 
2 0 superfluous display commands from the command queue (1 20, FIG. 3} begins at step 252 by 
identifying overridden commands. An overriding display command is essentially a display 
command that reverses or overrides an overridden di^lay command in the command queue 
that was earlier issued. For example, an earlier issued command can move an image of an 
aircraft to the right, and an overriding command later issued can move the image of the aircraft 
25 to the left by an equal amount. Overridden commands are also discussed in conjunction with 
FIG. 3 above. 

At step 254, redundant display commands in the command queue 120 are identified. A 
redundant display command is a diq)lay command that accomplishes no additional function in 
30 view of an earlier issued display command in the command queue 120. For example, an earlier 
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issued display command can specify that the color of an aircraft image is red, ajad a redundant 
command later issued can again specify that the im age of the airca^ is red. 

Superfluous display commands in the command queue are identified at step 256. A 
5 superfluous display command is a display command that is not valid in the given context For 
example, a display command that sets the color of an object and associated display image to white, 
when a default color associated with the object is white, has no puipose and is superfluous. 

In one particular embodiment, the search for overridden, redundant, and/or superfluous 
1 0 display commands is accelerated by using hash tables (e.g., only commands acting on the same 
object can be overridden or redundant display commands). 

At step 258, the overridden, redundant, and/or superfluous display commands are removed 
from the command queue 120. In this way, the conmiand queue 120 is miniraized in size. In one 
15 air traffic control system, the command queue size stabilizes at about 100 kilobytes (kB). . 

At step 260, the display commands remaining in the command queue 120 are kept in their 
original oider^ At st^ 262, it is indicated that new display commands received in the command 
queue are also kept in order of reception. 



20 



25 



Refening now to FIG. 7, an exemplary playback process 300 includes receiving, at step 
302, a request to play back an ATC diq)lay associated with a specified time of interest For an 
ATC system, for example, the request can be to display what happened at a specific time (e.g. 
"show what happened at 12:03:00."). 



Once the request is received, the system locates on the storage device (36, FIG. 1) a 
dynamic snapshot having a time stamp prior to the time of interest, and retrieves the dynamic 
snapshot at step 304. As described above, the dynamic snapshot coiresponds to a set of display 
commands representative of the state of images on the monitor 26 (FIG. 1), corresponding to 
3 0 the time prior to the time of interest. The time stamp preferably coiresponds to the closest 
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earlier time at which a dynamic snapshot was recorded, prior to the time of interest. 
Continuing witibi the above example^ assuming that a dynamic snapshot is stored in the storage 
device 36 every five (5) minutes beginning on the hour, then the desired dynamic snapshot i$ 
the dynamic snapshot stored at 12:00:00, having a corresponding time stamp. 

5 

At step 306, additional display commands are retrieved from the storage device 36. As 
described above in conjunction with FIG, 3, the additional display commands 154, 156, 158 are 
stored to the storage device between storage of successive dynamic snapshots 152, 160. The 
retrieved additional display commands, having time stamps, represent a group of display 
10 commands tbat occurred jErom the time of dynamic snapshot retrieved at step 304 up until the 
time of interest identified at step 3 02. 

It should be appreciated that the dynamic snapshot retrieved at stqp 304, in combination 
with the additional display commands retrieved at step 306, represent the* state of the real-time 
15 ATC diq>lay system 20 at the time of interest. Th^efore, the dynamic snapshot letrieved at 
step 304, in combination with the additional display commands retrieved at step 306, 
corresponds to an intemiediate dynamic sns^shot, which corresponds to the dynamic snapshot 
retrieved at step 304 moved forward in time, as described in conjunction with FIG. 2. 

20 In one particular embodiment, it is also possible to reduce the number of display 

commands firom among the retrieved dynamic snapshot and the retrieved additional display 
commands to remove overridden, redundant, and/or superfluous display commands in much the 
same way as described above for recording in conjunction with FIG. 6. 

2S At step 310, a display representative of the time of interest and conesponding to tbe 

intemiediate dynamic snapshot is generated on the playback ATC display system 50 (FIG. 1). 
In another embodiment, the di^lay representative of the time of interest is generated instead, or 
additionally, on the real-thne ATC display system 20 (FIG. 1). 

3 0 The exemplary playback method 300 of FIG. 7 essentially moves the dynamic snapshot 

24 
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forward in time and thus has an advantage of eliminating the need to play through data from the 
time of the retrieved dynamic snapshot to the time of interest. Instead, a display corresponding 
to the time of interest is provided. 

5 It should be appreciated that the playback described by the method 300 can be 

asynchronous fiom other aspects of the system 10 (FIG. 1). For example, the playback can be 
provided without impacting the real-time ATC display system 20 (FIG. 1), 

Referring now to FIG. 8, an alternate exemplary playback process 350 includes 
10 receiving at step 352 a request to play back an ATC display associated with a specified time of 
interest. For an ATC system, for example, the request can be to display what happened at a 
specific time (e.g- -"show what happened at 12!03:00-'0 

Once the request is received, the system locates on the storage device (36, FIG. 1) a' 
1 5 dynamic snapshot liaving a time stamp prior to the time of int^est> and retrieves the dynamic 
snapshot at step 354. As described above, the dynamic snapshot corresponds to a set of Ae 
display commands corresponding to state of images on the monitor 26 at a time prior to the 
time of interest The time stamp preferably corresponds to the closest earlier time at which a 
dynamic snapshot was recorded prior to the time of interest Continuing with the above 
20 example, assuming that a dynamic sn^shot is stored in the storage device 36 every five (5) 
mirmtes beginning on Ae hour, then the desired dynamic snapshot is the dynamic snapshot 
stored at 12:00:00, having a corresponding time stamp. 

At step 356 a display associated with the dynamic snapshot retrieved at step 354 is 
2 5 generated, for exan^le, on the playback ATC display system 50 of FIG. 1 . It shoxild be 

appreciated that the display thus generated does not correspond to the time of int^est pn>vided 
at step 352* In ano&er embodiment, the display associated with the dynamic snapshot retrieved 
at step 354 is generated instead, or additionaUy, on the real-time ATC display system 20. 

30 In order to move tfie display generated at step 356 forward in time, additional display 
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commands are retrieved from the storage device 36 at step 358. The retrieved additional 
di^lay commands coxrespond to those stored additional commands that correspond to times 
between the time of associated with the dynamic snapshot retrieved at step 354 and the time of 
int^st. 

5 

At step 360, the display generated at step 356 is played forward in time by applying the 
retrieved additional display commands. The display can be played forward either at a speed 
representing nonnal time progression, at a speed representing fast time progression, or a speed 
representing slow time progression. 

10 

Unlike the playback method of FIG. 7, the exemplary playback method of FIG. 8 allows 
. a user to view a generated display corresponding to any time between the time associated with 
the dynamic ^apshot retrieved at step 354 and the time of interest. In another embodiment, the 
user can also view a generated display corresponding to a time after the time of interest. 
15 • 

It should be appreciated that the playback described by the method 350 can be 
asynchronous from other aspects of the system 10 (FIG. 1). For example, the playback can.be 
provided without, inipacting the real-time ATC display system 20 (FIG. 1)- 

20 While the asynchronous storage and playback of a system sn^shot has been shown and 

described in association wit^ 2D scene graph display commands and wifli an ATC display, it 
should be appreciated, that in other embodiments, the display commands can include, but are 
not limited to, 3D scene graph display commands, and conventional **paint" display connnands 
associated wi& otha: types of graphical displays, or any combination of 2D, 3D and '"painf * 

2 5 display commands. 

Also, while the playback mefliod of FIGS. 7 and 8 has been shown and described to be 
a playback occurring some time later than the generation of a corresponding real-time display, 
it should be understood that the playback can occur with only a very short time between the 

3 0 playback and the actual real-time display. In this way, the asynchronous storage and playback 
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c^ability caa be used to provide a system snapshot that xiser can view almost at the same time 
as the real-time display, without impacting the real-time system operation. 

It should be appreciated that the ''dynamic snapshot** storage and playback technique of 
5 the present invention finds application in a number of different systems but is particularly well 
suited for object-oriented systems. However, any system that provides software commands or 
instructions, having overridden, redundant, or superfluous commands or instructions as 
described above, and for which such overridden, redundant, or superfluous commands can be 
eliminated so as to limit the size of a dynamic snapshot, is suitable for the above-described 

10 techniques. For example, Ihe above-described techniques can also be applied to a database 
system. It will be recognized that a database software applicaiton has sofhvare commands, 
which are suitable for removing ovemdden, redundant^ and superfluous commands in much the 
same way as described above in conjunction with FIGS. 3 and 6. For example, a "set" 
conmiand, used to place a number at a memory location can be overridden at a lat^ time by 

1 5 another "set" commarxd that places a number at the same location. Therefore, the techniques of 
the present invention can provide storage of dynamic snapshots, and playback of the system 
states in mudi the same was as described above 

The "dynamic snapshot" technique has a number of advantages including but not 

2 0 Umited to the following: (1) there is no need to serialize/de-serialize the objects themselves 

(only the commands); (2) the defeult object*s state does not need to be recorded; (3) if 
additional commands are recorded and time-stamped, then it is possible to playback the 
modification to the system from any dynamic snapshot; and (4) it is possible to seek very 
quickly by re-creating the dynamic snapshot at any particular time. 

25 

Having described preferred embodiments of flie invMtion it will now become apparait 
to diose of ordinary skill in the art that othar embodiments incorporating these concepts may be 
used. Additionally, the software included as part of the invention may be embodied in a 
compiitcr program product that includes a computer useable medium- For example, such a 

3 0 computer usable medium can include a readable memory device, such as a hard drive device, a 
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CD-ROM, a DVD-ROM, or a computer diskette, having comimter readable program code 
segments stored tibereon. The computer readable medium can also include a communications 
link, either optical, wired, or wireless, having program code segments carried thereon as digital 
or analog signals. Accordingly, it is submitted that that the invention should not be limited to 
5 the described CTibodiments but rather should be limited only by the spirit and scope of the 
appended claims. All publications and references cited herein are expressly incorporated 
herein by reference in their entirety. 

What is claimed is: 
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